Repository navigation
Conversation
lxy-9602
reviewed
Nov 6, 2026
| if (options.GetMergeEngine() != MergeEngine::DEDUPLICATE && | ||
| options.GetMergeEngine() != MergeEngine::PARTIAL_UPDATE) { | ||
| return Status::NotImplemented( | ||
| "PK realtime supports only the DEDUPLICATE and PARTIAL_UPDATE merge engines"); |
Member
There was a problem hiding this comment.
With partial-update and DVs enabled, pushing value predicates into high-level disk files can discard rows needed to complete an in-memory partial update.
For example:
Disk (after compaction): (id=1, a="old", b="keep")
Memory partial update: (id=1, a="new", b=null)
Query: WHERE a = 'new'
Expected: (id=1, a="new", b="keep")
Actual: (id=1, a="new", b=null)
The disk row is filtered out before merging, so b="keep" is lost.
Please either push down only primary-key predicates across all disk levels and evaluate value predicates after merging, or reject the combination of realtime partial-update and DVs for now.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Enable partial-update for primary-key realtime writes by reusing the existing merge functions and options used by the ordinary Writer.
Related issue (tracked separately): #524
PARTIAL_UPDATEalongsideDEDUPLICATE, preserving the existing deduplicate ignore-delete restrictions. Standalone aggregation and first-row engines remain unsupported.Known Issues
This PR aligns realtime partial-update merging with the existing ordinary Writer behavior by reusing the same merge functions. It does not fix the existing staged-merge defects or change non-realtime merge/read behavior. The ordinary Writer reproductions are tracked separately in #524 and will be addressed independently. Realtime may inherit these limitations; this PR does not establish correctness for all affected combinations.
Tests
Previously completed validation for feature commit
2e2f194ccdab76831b2988a9b749f5d8c299d8f7: 9 realtime integration tests and 20 core tests passed (29 total). These tests were not rerun when publishing this PR.The four new integration tests are:
RealtimeWriteInteTest.TestPkPartialUpdateDeleteAcrossCommitsRealtimeWriteInteTest.TestPkPartialUpdateSequenceGroupsRealtimeWriteInteTest.TestPkPartialUpdateAggregationSpillAndRecoveryRealtimeWriteInteTest.TestPkPartialUpdatePartitionedDefaultAggregationRetractThe other five integration tests cover existing primary-key reads, recovery, writer handoff, and multiple batches. Core coverage includes
KeyValueFileStoreWriteTest.*andFileStoreWriteTest.*.Coverage exercises real memory queries, prepare/commit and normal disk reads, spill, reopen, and replay. This is focused validation, not exhaustive partial-update semantics, full CI, Java interoperability, or all repository tests.
API and Format
No public
include/paimon/API, storage format, or protocol changes.Documentation
Updated the realtime user guide to document partial-update support and its scope.
Generative AI tooling
Generated-by: OpenAI Codex (CLI 0.160.0)